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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE _3_ MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

Responsive to communication(s) filed on August 23, 2001 through July 16. 2002 . 
2a)D This action is FINAL. 2b)£3 This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 1 1 , 453 O.G. 213. 
Disposition of Claims 

4) £3 Claim(s) 1-97 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1-97 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) £3 The specification is objected to by the Examiner. 

10) E1 The drawing(s) filed on 23 August 2001 is/are: a)S accepted or b)Q objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

11) D The proposed drawing correction filed on is: a)Q approved b)D disapproved by the Examiner. 

) If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

1 3) £3 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (f). 

a)QAII b)D Some*c)l3 None of: 

1 .03 Certified copies of the priority documents have been received. 

2.Q Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

1 5) S Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 1 20 and/or 1 21 . 
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1) |2Sl Notice of References Cited (PTO-892) 

2) Notice of Draftsperson's Patent Drawing Review (PTO-948) 
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4) O Interview Summary (PTO-41 3) Paper No(s). 

5) O Notice of Informal Patent Application (PTO-152) 

6) □ Other: 
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DETAILED ACTION 



Priority 



L It is noted that the PRELIMINARY AMENDMENT filed November 13, 2001 causes this 
application to contain a specific reference to international application No. PCT/BG00/00718 filed 
on March 1, 2000, published in English under PCT Article 21(2), as the first sentence of the 
specification in order to rely on the filing date of the prior application under 35 U.S.C. 120. 



applications filed in the United Kingdom on 05 March 1999. It is noted, however, that applicant 
has not filed certified copies of the applications as required by 35 U.S.C. 1 19(b). A certified copy 
of any foreign priority document must be provided by the applicant if the parent international 
application has not entered the national stage under 35 U.S.C. 371 (the photocopy received from 
the International Bureau cannot be used). If the parent international application has entered the 
national stage under 35 U.S.C. 371, the applicant, in the continuing application, may state that the 
priority document is contained in the national stage application. See MPEP § 1893 .03(f). 

3. The Examiner notes that this application file does not contain evidence that a claim for 
foreign priority was made in the international application. 



2. 



Acknowledgment is made of applicants claim for foreign priority based on five 



4. A search report of the European Patent Office concerning document GBA 9925561 is 
present in the application file, and it has been considered by the Examiner. 
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Information Disclosure Statement 



5. The information disclosure statement filed August 23, 2001 (paper 6) fails to comply with 
37 CFR 1.98(a)(1), which requires a list of all patents, publications, or other information 
submitted for consideration by the Office. Because its defects appear to be listing errors, as 
inaccuracy in recording the author, title, date, or pages, the Examiner has changed citations on the 
disclosure statement to correspond to the copies provided. If the changes and/or additions are 
acceptable to the Applicant, no action is required. Further submissions should comply with 37 
CFR 1.97 and 37 CFR 1.98 as of the date of their submission. 

6, The information disclosure statement filed December 4, 2001 (paper 7) and copies of five 
applications that were filed May 30, 2001 are present in the application file, and they have been 
considered by the Examiner. 



7. The title is objected to because it is not sufficiently descriptive of the invention. A new 
title is required that is clearly indicative of the invention to which the claims are directed. See 
MPEP § 606.01. The Examiner suggests that the Applicant consider a title including these 
elements: "Database Annotation and Retrieval Including Phoneme Data " 



Specification 



8. The specification is objected to as failing to provide proper antecedent basis for the 
claimed subject matter. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). Appropriate correction is 
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Art Unit: 2654 

required. The specification should teach the following claimed subject matter: data arranged in 
blocks of equal time duration (claim 4 and others). 

The Examiner was unable to find any description related to this subject matter in the 
specification. Any feature of the invention should be apparent in the descriptive portion of the 
specification with clear disclosure as to its import, and where possible, it should be identified in 
the descriptive portion of the specification by reference to the drawing, designating the item or 
items therein to which the term applies. This is necessary in order to insure certainty in construing 
the claims in the light of the specification. To avoid an objection for adding new matter to the 
specification, the Applicant should point out specific support in the disclosure as filed for any 
added description. 

9. The specification is objected to as failing to provide proper antecedent basis for the 
claimed subject matter. The specification should clearly teach the following claimed subject 
matter; "parol" (claim 10 and others). 

The specification does not use this terminology. While the Applicant is not limited to the 
nomenclature used in the application as filed, the claim terminology should have clear support or 
antecedent basis in the specification for the terms appearing in the claims. The meaning of every 
term used in any of the claims should be apparent from the descriptive portion of the specification 
with clear disclosure as to its import, and where possible, it should be identified in the descriptive 
portion of the specification by reference to the drawing, designating the item or items therein to 
which the term applies. This is necessary in order to insure certainty in construing the claims in 
the light of the specification. See 37 CFR 1.75(d)(1) and MPEP § 608.01(o). 
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If the Applicant is going to maintain the terminology "parol", the Examiner suggests that 
the Applicant ensure that the definition and antecedence in the specification clearly establish the 
terminology so that the scope of the claimed invention is not indefinite. To avoid an objection for 
adding new matter to the specification, the Applicant should point out specific support in the 
disclosure as filed for any added description. 



Claim Informalities 

10. Claim 5, and by dependency claims 14-23 and 96-97, are objected to under 37 CFR 1 .75(a) 
because the meaning of the phrase "said database" needs clarification. Because no database was 
previously said, it is unclear as to what element this phrase is making reference. To further timely 
prosecution and evaluate prior art, the Examiner has interpreted this phase to refer to -a 
database—. 



1 1 . Claim 10, and by dependency claims 14-23 and 96-97, are objected to under 37 CFR 
1.75(a) because the meaning of the word "parol" needs clarification. The term is so different from 
that which is generally accepted in the art to which this invention pertains that the Applicant 
should provide a definition, clarification, and correlation with art-accepted terminology. See 
37 CFR 1.71 and MPEP 608.01. 

Dictionary definitions of which the Examiner is aware provide related, but varying, usage 
of the word "parol " They also indicate that the word is jargon in arts that do not pertain to this 
invention, an obsolete spelling, or not currently used. To further timely prosecution and evaluate 
prior art, the Examiner has interpreted this phase to refer to -phoneme and/or word data-. 
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The rule that an Applicant can act as his own lexicographer presupposes that nonstandard 
terminology is clearly detailed in the disclosure. Appropriate correction is required. No new 
matter may be added. 

12. Claim 22 is objected to under 37 CFR 1.75(a) because the meaning of the phrase "said step 
of generating phoneme and word data" needs clarification. The previously said generating step 
generates phoneme and/or word data. It is unclear as to what "phoneme and word" element this 
phrase is making reference. To further timely prosecution and evaluate prior art, the Examiner has 
interpreted this phase to refer to —phoneme and/or word data--. 

13. Claim 23 is objected to for the same reasons as claim 22 because the limitations are recited 
using obviously similar phrases. 

14. Claim 32 is objected to for the same reasons as claim 22 because the limitations are recited 
using obviously similar phrases. 

15. Claim 33 is objected to for the same reasons as claim 22 because the limitations are recited 
using obviously similar phrases. 



16. Claim 46 is objected to for the same reasons as claim 10 because the limitations are recited 
using obviously similar phrases. 
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17. Claim 47 is objected to for the same reasons as claim 10 because the limitations are recited 
using obviously similar phrases. 

18. Claim 71 is objected to for the same reasons as claim 10 because the limitations are recited 
using obviously similar phrases. 

19. Claim 72 is objected to for the same reasons as claim 10 because the limitations are recited 
using obviously similar phrases. 

20. Claim 8 1 is objected to under 37 CFR 1 75(a) because the meaning of the phrase "said step 
of converting words into phonemes" needs clarification. Because the previously recited 
converting step converts to phoneme data, not to phonemes, it is unclear as to what element this 
phrase is making reference. To further timely prosecution and evaluate prior art, the Examiner has 
interpreted this phase to refer to —phoneme data--. 

21. The Examiner notes, without objection, that the dependency of claim 2 is to any preceding 
claim. Since cancellation of claims during prosecution, addition of claims, and renumbering of 
claims for printing of allowed applications can result in rearrangement of the sequence of claims, 
the Applicant is encouraged to reword the dependency limitation of the claim during normal 
review and revision of the disclosure. 



22. The Examiner notes, without objection, that the dependency of claim 14 is to any 
preceding claim. Since cancellation of claims during prosecution, addition of claims, and 



1 . 



ii 
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renumbering of claims for printing of allowed applications can result in rearrangement of the 
sequence of claims, the Applicant is encouraged to reword the dependency limitation of the claim 
during normal review and revision of the disclosure. 



23. The Examiner notes, without objection, that the following phrases assume inherent 
antecedence by implication to previously recited claim elements: (claim 15) "the user's"; (claim 
16) "the word search"; (claim 16) "the phoneme search"; (claim 18) "said phoneme search"; 
(claim 18) "the phoneme sequence"; (claim 20) "said phoneme search"; (claim 22) "the user"; 
(claim 25) "the user's"; (claim 26) "the phoneme search"; (claim 28) "said phoneme search"; 
(claim 28) "the phoneme sequence"; (claim 28) "the user's"; (claim 30) "said phoneme search"; 
(claim 32) "the user"; (claim 34) "the decoded words"; (claim 39) "the words"; (claim 53) "the 
word data". In claims 16 and 26, the selected portions of the database appear to imply the selected 
portions of the lattice in a preceding claim, which is the data of preceding claims, of which the 
database is comprised. The Applicant may wish to consider if the phrases recite the claimed 
subject matter that the Applicant wants. 



24. The Examiner notes, without objection, the possibility of informalities in the claims. The 
Applicant may wish to consider changes during normal review and revision of the disclosure. 
In claim 5, should the word "blocks" be —block's—? 



25. The form of the claims does not follow Office practice. While there is no set statutory 
form for claims, the present Office practice is to insist that each claim must be the object of a 
sentence starting with "I (or we) claim", "The invention claimed is" (or the equivalent). The 
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Applicant is encouraged to insert a desired introduction before claim 1 . If, at the time of 
allowance, appropriate terminology is not present, it is inserted by the technical staff. See 
MPEP § 608.01(m). 



Claim Rejections - 35 USC § 112 

26. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

27. Claims 73, 74, 77, and 78 are rejected under 35 U S C. 1 12, second paragraph, as being 
indefinite for failing to particularly point out and distinctly claim the subject matter which 
applicant regards as the invention. 

28. Regarding claim 73, the phrase "the phonemes associated with said links" (page 40, line 
29) lacks clear antecedent basis in the claim. No links were previously said, and no phonemes 
have yet been associated with any links. To further timely prosecution and evaluate prior art, the 
Examiner has interpreted this phase to refer to -phonemes--. 

29. Regarding claim 74, the phrase "the words associated with said links" (page 40, lines 32- 
33) lacks clear antecedent basis in the claim. No links were previously said, and no words have 
yet been associated with any links. To further timely prosecution and evaluate prior art, the 
Examiner has interpreted this phase to refer to -words-. 




Application/Control Number. 



>34, 799 



PAGE 10 



Art Unit: 2654 

30. Claim 77 is indefinite in the same way as claim 73 because the limitations are recited using 
obviously similar phrases. 

31. Claim 78 is indefinite in the same way as claim 74 because the limitations are recited using 
obviously similar phrases. 

Claim Rejections - 35 USC §101 

32. The following is a quotation of 35 U.S.C 101 : 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any 
new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of 
this title. 

33. Claims 1-13 and 93-97 are rejected under 35 US.C. 101 because the claimed invention is 
directed to non-statutory subject matter. . 

34. Regarding claim 1, data, nodes, and links of a lattice comprise only nonfunctional 
descriptive material. Data per se comprises only nonfunctional descriptive material. 
Nonfunctional descriptive material includes the compilation or mere arrangement of data. Data 
associating phonemes and at least one word with links of a lattice merely provides an arrangement 
of the data compilation. Purely nonfunctional descriptive material is nonstatutory despite the fact 
that it might inherently have some usefulness. Such descriptive material is not statutory because it 
is neither a useful process, machine, manufacture, nor composition of matter. Purely 
nonfunctional descriptive material cannot alone provide a practical application. All claim 
limitations have been considered, and the claimed data has been found nonstatutory as a mere 
arrangement of data 
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a. The farther limitations of the dependent claims 2-13 continue to describe the 
arrangement of data, and do not provide any statutory category of invention having a practical 
application to satisfy the requirements of 35 U.S.C 10L 

b. The further limitations of dependent claim 96 are not limited to processor- 
implementable instructions, but the alternative claimed invention may instead include only data 
according to claim 1, which is nonfunctional descriptive material When nonfunctional descriptive 
material is recorded on a computer-readable medium, it is not structurally and functionally 
interrelated to the medium, but merely carried by the medium. However, claim 96 does not claim 
a medium. The data carrier of claim 96 may be a further arrangement of data; however, is appears 
to be functional data, inasmuch as the data of the data carrier carries other data. Functional data is 
not statutory because it is neither a useful process, machine, manufacture, nor composition of 
matter All claim limitations have been considered, and the claimed data carrier carrying data has 
been found nonstatutory as a mere arrangement of data. The claim does not recite any structural 
medium, and it does not recite any data structurally and functionally interrelated to the medium. 
The further limitations of this claimed embodiment, dependent to claim 1, continue to describe the 
arrangement of data of claim 1. These limitations do not provide the necessary structure and 
functional and structural interrelationship to satisfy the requirements of 35 U.S.C. 101. 

c. The further limitations of dependent claim 97 are directed to a program of 
instructions, A list of instructions comprises only functional descriptive material. The description 
of instructions is not a physical article, nor is it a statutory process, as it is not acts being 
performed. Instructions that are not encoded on a computer-readable medium do not define both 
functional and structural interrelationships between the instructions and the medium, which permit 
the instructions' functionality to be realized. Functional descriptive material that is not claimed as 
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embodied in computer-readable media is descriptive material per se and is not statutory because it 
is neither a useful process, machine, manufacture, nor composition. Functional descriptive 
material on computer-readable media carries out an algorithm that electrically changes a general 
purpose computer into a special purpose machine by activating electrical paths and deactivating 
other paths. All claim limitations and have been considered, and the matter of claim 97 has been 
found nonstatutory as a computer program per se, because the underlying process describing the 
instructions does not provide them with the necessary functional and structural interrelationship 
with any medium to satisfy the requirements of 35 U.S.C. 101. 

35. Regarding claim 93, data, audio data, annotation data, and phoneme data comprise only 
nonfunctional descriptive material. Data per se comprises only nonfunctional descriptive material. 
Nonfunctional descriptive material includes the compilation or mere arrangement of data. Data 
associating annotation data and audio data merely provides an arrangement of the data 
compilation. Purely nonfunctional descriptive material is nonstatutory despite the fact that it 
might inherently have some usefulness. Such descriptive material is not statutory because it is 
neither a useful process, machine, manufacture, nor composition of matter. Purely nonfunctional 
descriptive material cannot alone provide a practical application. All claim limitations have been 
considered, and the claimed data has been found nonstatutory as a mere arrangement of data 



36. Regarding claim 94, data, video data, audio data, annotation data, and phoneme data 
comprise only nonfunctional descriptive material. Data per se comprises only nonfunctional 
descriptive material. Nonfunctional descriptive material includes the compilation or mere 
arrangement of data. Data associating annotation data or video data and audio data merely 
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provides an arrangement of the data compilation. Purely nonfunctional descriptive material is 
nonstatutory despite the fact that it might inherently have some usefulness. Such descriptive 
material is not statutory because it is neither a useful process, machine, manufacture, nor 
composition of matter. Purely nonfunctional descriptive material cannot alone provide a practical 
application. All claim limitations have been considered, and the claimed data has been found 
nonstatutory as a mere arrangement of data 

37. Regarding claim 95, data, text data, annotation data, and phoneme data comprise only 
nonfunctional descriptive material. Data per se comprises only nonfunctional descriptive material. 
Nonfunctional descriptive material includes the compilation or mere arrangement of data. Data 
associating annotation data and text data merely provides an arrangement of the data compilation. 
Purely nonfunctional descriptive material is nonstatutory despite the fact that it might inherently 
have some usefulness. Such descriptive material is not statutory because it is neither a useful 
process, machine, manufacture, nor composition of matter. Purely nonfunctional descriptive 
material cannot alone provide a practical application. All claim limitations have been considered, 
and the claimed data has been found nonstatutory as a mere arrangement of data 



Claim Rejections - 35 USC §102 
38. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale 
in this country, more than one year prior to the date of application for patent in the United States. 
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BirdetaL 

39. Claims 1-8, 10, and 12-13 are rejected under 35 U.S.C. 102(b) as being anticipated by 
Steven Bird and Mark Liberman, "Towards a Formal Framework for Linguistic Annotations," 
( Bird et al l already of record. 



40. Regarding claim 1, Bird et al. describes the claimed limitations as a whole recognizable to 
one versed in the art by explicitly describing the following elements: 

data defining nodes within a lattice [at section 3. 1, as lines of numbers and labels in the 
corpus interpreting times as nodes in a directed, acyclic graph]; 

data defining links connecting the nodes within a lattice [at section 3.1, as lines of labels 
and numbers in the corpus interpreting labels as edges which connect nodes in a directed, acyclic 
graph]; 

data associating phonemes with the links [at section 3 . 1, as type of the edge label P for 
phonetic transcription]; and 

data associating words with links [at section 3 . 1, as type of the edge label W for word]. 



41 . Regarding claim 2, Bird et al. also describes: 

the data is arranged in blocks of nodes [at section 3.2, as different tiers for the description 
similar to TIMIT, chunks, fragments, or contiguous sequences of anchors, say dialog acts]. 



42. Regarding claim 3, Bird et al. also describes: 

time stamp data for each node [at section 3.1, as times on each line interpreted as nodes]. 
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43. Regarding claim 4, Bird et al. also describes: 

arranging in blocks of equal time duration [at section 3.1 and Fig. 1, as, for example, the 
block 3/5200 -> 4/6100 -> 5/8720 -> 6/9680 equals the block 3/5200 -> 6/9680 in duration]. 



44. Regarding claim 5, Bird et al. also describes: 

data defining each block's location within a database [at sectioii^87 as offsets in time- 
function files that provide the time line to annotations including time references]. 



45. Regarding claim 6, Bird et al. also describes: 

associated data defining a time sequential signal [at section 4.8, as the time line via a 
sampling rate or some other mechanism]; and 

the time stamp information is synchronized with the time sequential signal [at section 4T§ 
as the annotations* time references are indexed with the offsets of time-function files, and perhaps 
mediated for the granularity]. 



46. Regarding claim 7, Bird et al. also describes: 

the associated data defines audio and/or video [at section 3.1, as time measured in audio 
samples]. 



47. 
and 



Regarding claim 8, Bird et al. also describes: 

the associated data defines speech data [at section 3 .1, as read speech of the database files]; 
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from which the data defining the lattice is derived [at section 3 . 1, as lines of labels and 
numbers in the corpus from the associated data lines of the files are interpreted as a directed 
acyclic graph]. 



48. Regarding claim 10, Bird et al. also describes: 

the speech data defines the parol of a plurality of speakers [at sections 3 .1 and 3.2, as read 
speech of exemplary female speaker 'jspO' from the TIMIT corpus and the Partitur speech signal 
fragment under discussion from the Verbmobil corpora]; and 

the data defines a separate phoneme and word lattice for the parol of each speaker [at 
sections 3.1 and 3.2 and Figs. 1 and 2, as the example file and acyclic graph for female speaker 
'jspO 5 from TIMIT and the example Partitur annotations lines expressed as in Figure 2]. 



49. Regarding claim 12, Bird et al. also describes: 

a node connected to a plurality of other nodes by a plurality of links [section 3 .2 and Fig. 2, 
as a unit on a tier with links to units on different tiers]. 



50. Regarding claim 13, Bird et al. also describes: 

one of the links is associated with a phoneme [section 3 .2 and Fig. 2, as links on the 
phonetic segment tier (M) with links to units on other tiers]; and 

one of the links is associated with a word [section 3.2 and Fig. 2, as links on the 
orthography tier (O) with links to units on other tiers]. 



Application/Control Number. 
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Kupiec 

51. Claims 35, 60, 84, 86, 87, and 89 are rejected under 35 U.S.C. 102(b) as being anticipated 
by Kupiec [US Patent 5,500,920]. 

52. Regarding claim 35, Kupiec [at column 27, line 19-column 28, line 10] describes the 
claimed limitations as a whole recognizable to one versed in the art by explicitly describing the 
following elements: 

a text to phoneme converter for generating phoneme data for text in the data file [at column 
21, lines 8-14, as techniques for automatically constructing a phonetic index from orthographic 
spellings in a corpus of text]; 

generating means for generating annotation data by combining the phoneme data and 
words in the text data [at column 29, line 58-column 30, line 44, as a technique for constructing a 
phonetic index containing a vector of orthographic spellings from the text corpus and phone 
sequences represented by a network that provides indexes into the vector]. 

53. Claim 60 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 35. Because Kupiec describes the similar limitations as 
indicated there, this claim thus is anticipated accordingly. 



54. Regarding claim 84, Kupiec [at column 27, line 19-column 28, line 10] describes the 
claimed limitations as a whole recognizable to one versed in the art by explicitly describing the 
following elements: 
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generating phoneme and word data for an input query [at column 9, lines 22-51, as convert 
into phones and words the user's utterance spoken into the audio transducer]; 

and searching using it [at column 1 1, lines 40-41, as executing the query in the IR 
subsystem]; 

in the phoneme and word lattice [at column 12, line 58-column 13, line 36, as search for 
matching in the phonetic index represented as a network and orthographic vector]; 

outputting search results of the search [at column 12, lines 34-46, as presenting the results 
with the matched search terms to the user]. 



55. Regarding claim 86, Kupiec [at column 27, line 19-column 28, line 10] describes the 
claimed limitations as a whole recognizable to one versed in the art by explicitly describing the 
following elements: 

combining a data file with annotation data corresponding to a data file, the annotation data 
including phoneme data [at column 29, line 60-column 30, line 44, as a phonetic index containing 
phone sequences constructed with the contents of the text corpus]; and 

storing the data file with the annotation data [at column 5, lines 43-47 and Fig. items 40- 
42, as a corpus and the word index that it uses are in the IR subsystem in the system]. 



56, Claim 87 sets forth limitations similar to limitations set forth in claim 84. Kupiec 
describes the limitations as indicated there. Kupiec also describes further additional limitations as 
follows: 

means for generating [at column 9, lines 22-51, as a speech recognition transcription 
technique]; 
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means for searching [at column 1 1, lines 40-41, as the IR subsystem]; and 
means for outputting [at column 12, lines 34-46, as processor in conjunction with visual 
display]. 

57. Claim 89 sets forth limitations similar to limitations set forth in claim 86. Kupiec 
describes the limitations as indicated there. Kupiec also describes further additional limitations as 
follows: 

means for inputting [at column 29, lines 51-57, as code for reading]; 
means for storing [at column 5, lines 43-47 and Fig. items 40-42, as the ER subsystem in 
the system]. 



Ellozv 

58. Claims 93-95 are rejected under 35 U.S.C 102(b) as being anticipated by Ellozy [US 
Patent 5,649,060]. 



59. Regarding claim 93, Ellozv describes the claimed limitations as a whole recognizable to 
one versed in the art by explicitly describing the following elements: 
audio data [at column 5, line 34, as audio data]; 

annotation data corresponding to the audio data [at column 5, lines 33-34, as decoded text 
aligned with the audio data]; 

and including phoneme data [at column 9, lines 18-24, as the phonetic string of the 
decoded text]. 
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60. Regarding claim 94, Ellozy describes the claimed limitations as a whole recognizable to 
one versed in the art by explicitly describing the following elements: 
video data [at column 7, line 10, as video data]; 

audio data corresponding to the video data [at column 4, line 54, as audio data in the 
audio/video data]; 

annotation data corresponding to the audio data [at column 5, lines 33-34, as decoded text 
aligned with the audio data]; 

and including phoneme data [at column 9, lines 18-24, as the phonetic string of the 
decoded text]. 



61. Regarding claim 95, Ellozy describes the claimed limitations as a whole recognizable to 
one versed in the art by explicitly describing the following elements: 
text data [at column 6, line 62, as transcript in text store]; 

annotation data corresponding to the text data [at column 6, lines 63-65, as decoded text 
matched to the transcript]; 

and including phoneme data [at column 9, lines 18-24, as the phonetic string of the 
decoded text]. 



Claim Rejections - 35 USC §103 
62. The following is a quotation of 35 U S C. 103(a) which forms the basis for all obviousness 
rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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63. This application currently names joint inventors. In considering patentability of the claims 
under 35 U.S.C. 103(a), the examiner presumes that the subject matter of the various claims was 
commonly owned at the time any inventions covered therein were made absent any evidence to 
the contrary. Applicant is advised of the obligation under 37 CFR 1 .56 to point out the inventor 
and invention dates of each claim that was not commonly owned at the time a later invention was 
made in order for the examiner to consider the applicability of 35 U.S.C. 103(c) and potential 35 
U.S.C. 102(e), (f) or (g) prior art under 35 U.S.C. 103(a). 

Bird et aL and Kupiec 

64. Claims 9, 14, 21-24, 31-34, 36-47, 50-51, 54-59, 61-72, 75-76, 79-83, 85, 88, and 96-97 
are rejected under 35 U.S.C. 103(a) as being unpatentable over Steven Bird and Mark Liberman, 
"Towards a Formal Framework for Linguistic Annotations," ( Bird et al. ), already of record, in 
view of Kupiec [US Patent 5,500,920]. 



65. Regarding claim 9, Bird et al. also describes: 

the speech data as audio [at section 3. 1, as read speech measured in audio samples]. 

Bird et al. [at section 1] also describes software for the creation of the database. However, 
Bird et al. does not further describe the software that should be used. In particular, Bird et al. does 
not explicitly describe automatic speech recognition. 

Kupiec [at column 13, lines 26-29] describes a network embodiment of a phonetic index 
with similarities to the one of Bird et aL and Kupiec also describes: 

speech data as audio [at column 9, line 19, as speaking into an audio transducer]; 

defining the lattice through automatic recognition [at column 21, lines 12-14, as the 
phonetic index can be constructed automatically]; 
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and defining phonemes and words by passing an audio signal through an ASR system [at 
column 9, lines 22-51, as convert into phones and words the user's utterance spoken into the audio 
transducer using a speech recognition transcription technique]; 

where a lattice defines the words and phonemes [at column 13, lines 26-29, as a vector 
storing orthographic spellings and a network representing phones]. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, but uses the speech recognition system for conversion of the 
input audio signal to phonemes and words. Bird et aL on the other hand, provides an audio 
corpus of speech signals, but does not describe the software to be used to convert the speech to 
phonemes and words. In view of the similarities between Bird et aL and Kupiec , an artisan 
seeking to implement Bird et al. 's automatic generation of phonemes and words would have found 
it obvious at the time of invention to use the concept of speech recognition as provided by Kupiec 
to create the word and phoneme lattice of Bird et al. because Kupiec converts an audio signal 
comprising speech to the phonemes and words that can populate Bird et al. 's lattice, just as Kupiec 
places them in a vector and corresponding network. 



66. Regarding claim 14, Bird et al. also describes: 

generating phoneme and/or word data for an input query [at section 5.3, as query syntax of 
type P and/or W]; 

and searching the lattice using it [at section 5.3, as operating the query over the annotation 
data for a match]. 

Although Bird et al. [at section 5.3] returns a result from matching the annotation data, 
Bird et al. does not explicitly describe outputting the search results. 
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Kupiec [at column 13, lines 26-29] describes a network embodiment of a phonetic index 
with similarities to the one of Bird et aL and Kupiec also describes: 

generating phoneme and/or word data for an input query [at column 9, lines 18-21, as 
converting an input question into a phonetic transcription]; 

and searching using it [at column 11, lines 40-41, as executing the query in the IR 
subsystem]; 

in the phoneme and word lattice [at column 12-line 58-column 13, line 36, as search for 
matching in the phonetic index represented as a network and orthographic vector]; 

outputting search results of the search [at column 12, lines 34-46, as presenting the results 
with the matched search terms to the user]. 

To the extent that Kupiec ' s phonetic network and orthographic vector is not innately a 
phoneme and word lattice, it would have been obvious to one of ordinary skill in the art of 
searching queries through a database at the time of invention to include Bird et al. 's word and 
phoneme lattice framework with Kupiec ' s concept of a phonetic network representing words in 
the vector because that would simplify the annotation framework to a single structure and allow 
phoneme and/or word searching to be combined in Kupiec 's search parameters. 

Because both Bird et al. and Kupiec search the annotation data for phonetic and/or word 
matches to an input query, and both identify and return search results relevant to the query, it 
would have been obvious to one of ordinary skill in the art of searching queries through a database 
at the time of invention to include Kupiec 's concept of outputting the search results to a user 
because information returned should be useful to the user in answering the question that the user 
posed to the system. 
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67. Regarding claim 21, Kupiec also describes: 

the search results are output to a display [at column 12, lines 34-46, as results relevant to 
matched search terms can be presented with visual display]. 



68. Regarding claim 22, Kupiec also describes: 

input query by the user is by voice [at column 9, lines 18-21, as the user inputs a question 
by speaking]; and 

generating phoneme and/or word data employs ASR [at column 9, lines 22-51, as a 
transcriber produces a phonetic transcription according to speech recognition]. 



69. Regarding claim 23, Kupiec also describes: 

the input query is typed [at column 1, lines 48-50, as user input is typically entered by a 
keyboard]; 

generating phoneme and/or word data employs text-to-phoneme conversion [at column 21, 
lines 8-14, as construct a phonetic index from orthographic spellings for any text]. 



70. Claim 24 sets forth limitations similar to limitations set forth in claim 14. Bird et al. and 
Kupiec describe and make obvious the limitations as indicated there. Kupiec also describes 
further additional limitations as follows: 

means for generating phonemes and/or words [at column 9, line 20, as the transcriber]; 

means for searching [at column 11, lines 1 1-12, as the IR subsystem]; 

means for outputting [at column 12, lines 34-35, as the processor in conjunction with 
visual display]. 
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71 . Claim 3 1 sets forth additional limitations similar to limitations set forth in claim 21. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

72. Claim 32 sets forth additional limitations similar to limitations set forth in claim 22. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. Kupiec also 
describes: 

a word decoder [at column 11, lines 9-39, as the hypothesis generator and query 
constructor]. 

73. Claim 33 sets forth additional limitations similar to limitations set forth in claim 23. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

74. Regarding claim 34, Bird et al. and Kupiec describe and make obvious the claimed 
limitations as a whole recognizable to one versed in the art because Kupiec [at column 13, lines 
26-29] describes an embodiment of phoneme data represented by a network, and Kupiec explicitly 
describes the following elements: 

an ASR system for generating phoneme data for audio data [at column 9, lines 22-51, as a 
speech recognition transcription technique converts into phones the user's utterance spoken into 
the audio transducer]; 

a word decoder for identifying possible words in that phoneme data [at column 9, lines 38- 
48, as a hypothesis generator to determine candidate words in the phonetic transcription]; 
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generating means for generating data by combining that phoneme data and decoded words 
[at column 13, lines 4-39, as the hypothesis generator develops phone sequences represented by a 
network with indices into an orthographic vector]. 

Kupiec describes ASR for the user's input query, but Kupiec does not explicitly describe 
audio data in the data file for ASR to produce annotation data. 

Bird et al. [at section 3.1 and Fig. 2] also describes a network embodiment of a phonetic 
index with similarities to the one of Kupiec , and B ird et al. also describes:: 

the speech data as audio [at section 3.1, as read speech measured in audio samples]; and 

generating annotation data [at section 3 . 1, as interpreting each line of the TEVHT audio 
corpus to construct an annotation graph with phonetic transcription and orthographic word edge 
labels]. 

Bird et al. [at section 1 ] also describes software for the creation of the database. However, 
Bird et al. does not further describe the software that should be used. In particular, Bird et al. does 
not explicitly describe automatic speech recognition. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, but uses the speech recognition system for conversion of the 
input audio signal to phonemes and words. Bird et al , on the other hand, provides an audio 
corpus of speech signals, but does not describe the software to be used to convert the speech to 
phonemes and words. In view of the similarities between Bird et al. and Kupiec , an artisan 
seeking to implement Bird et al. 's automatic generation of annotation data having phonemes and 
words would have found it obvious at the time of invention to use the concept of speech 
recognition as provided by Kupiec to create the word and phoneme lattice of Bird et al. because 
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Kupiec converts an audio signal comprising speech to the phonemes and words that can populate 
Bird et al. 's lattice, just as Kupiec places them in a vector and corresponding network. 



75. Regarding claim 36, Bird et al. and Kupiec describe and make obvious the claimed 
limitations as a whole recognizable to one versed in the art because Kupiec [at column 13, lines 
26-29] describes an embodiment of phoneme data represented by a network, and Kupiec explicitly 
describes the following elements: 

input means for receiving an input voice signal [at column 9, lines 22-51, as the user's 
utterance spoken into the audio transducer]; 

speech recognition means for converting the input voice signal into phoneme data and 
words [at column 9, lines 22-51, as a speech recognition transcription technique converts the 
user's utterance spoken into the audio transducer into phones and words]; 

generating means for generating annotation data by combining the phoneme data and the 
words [at column 30, lines 38-44, as a technique for constructing a phonetic index containing a 
vector of orthographic spellings and phone sequences represented by a network that provides 
indexes into the vector]. 

Bird et al. [at section 3 .1 and Fig. 2] also describes a network embodiment of a phonetic 
index with similarities to the one of Kupiec , and B ird et al. also describes:: 

the speech data as audio [at section 3.1, as read speech measured in audio samples]; and 

generating annotation data [at section 3. 1, as interpreting each line of the TIMIT audio 
corpus to construct an annotation graph with phonetic transcription and orthographic word edge 
labels]. 
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Bird et al. [at section 1] also describes software for the creation of the database. However, 
Bird et al. does not further describe the software that should be used. In particular, Bird et al. does 
not explicitly describe automatic speech recognition. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, but uses the speech recognition system for conversion of the 
input audio signal to phonemes and words. Bird et al , on the other hand, provides an audio 
corpus of speech signals, but does not describe the software to be used to convert the speech to 
phonemes and words. In view of the similarities between Bird et al. and Kupiec , an artisan 
seeking to implement Bird et al. 's automatic generation of annotation data having phonemes and 
words would have found it obvious at the time of invention to use the concept of speech 
recognition of Kupiec ' s input via an audio transducer to create words and phonemes that could 
populate Bird et al. 's lattice from the user's input because with that capability the user may add 
personalized annotation to the phoneme and word lattice that annotates the corpus. 



76. Regarding claim 37, Bird et al. and Kupiec describe and make obvious the claimed 
limitations as a whole recognizable to one versed in the art because Kupiec [at column 13, lines 
26-29] describes an embodiment of phoneme data represented by a network, and Kupiec explicitly 
describes the following elements: 

input means for receiving a typed input from a user [at column 1, lines 48-50, as user input 
is typically entered by a keyboard]; 

converting means for converting words into phoneme data [at column 21, lines 8-14, as 
techniques for automatically constructing a phonetic index from orthographic spellings in text]; 
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generating means for generating annotation data by combining the phoneme data and 
words in typed data [at column 29, line 58-column 30, line 44, as a technique for constructing a 
phonetic index containing a vector of orthographic spellings from the text corpus and phone 
sequences represented by a network that provides indexes into the vector]. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, and describes keyboard input as known for typical input of 
text by a user. Bird et aL on the other hand, provides an audio corpus of speech signals, but does 
not describe the software to be used to convert the speech to phonemes and words. In view of the 
similarities between Bird et al. and Kupiec , an artisan seeking to implement Bird et al/ s automatic 
generation of annotation data having phonemes and words would have found it obvious at the time 
of invention to use Kupiec 3 s concept of text to phoneme generation of Kupiec 3 s input via 
keyboard to create words and phonemes that could populate Bird et al. 's lattice from the user's 
input because with that capability the user may add personalized annotation to the phoneme and 
word lattice that annotates the corpus. 



77. Regarding claim 38, Bird et al. and Kupiec describe and make obvious the claimed 
limitations as a whole recognizable to one versed in the art because Kupiec [at column 13, lines 
26-29] describes an embodiment of phoneme data represented by a network, and Kupiec explicitly 
describes the following elements: 

means for receiving image data representative of text [at column 23, lines 24-28, as an 
input transducer for receiving an optically scanned document or other sequence of words]; 

character recognition means [at column 4, lines 48-49, as optical character recognition]; 
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for converting the image data into text data [at column 23, lines 30-32, as a processor 
transcribes the signal into orthographically represented words]; 

converting means for converting words in the text data into phoneme data [at column 21, 
lines 8-14, as techniques for automatically constructing a phonetic index from orthographic 
spellings in text]; 

generating means for generating annotation data by combining the phoneme data and 
words in typed data [at column 29, line 58-column 30, line 44, as a technique for constructing a 
phonetic index containing a vector of orthographic spellings from the text corpus and phone 
sequences represented by a network that provides indexes into the vector]. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, and describes optical input of document images for input of 
text by a user. Bird et aL on the other hand, provides an audio corpus of speech signals, but does 
not describe the software to be used to convert the speech to phonemes and words. In view of the 
similarities between Bird et al. and Kupiec , an artisan seeking to implement Bird et al.'s automatic 
generation of annotation data having phonemes and words would have found it obvious at the time 
of invention to use Kupiec 's concept of text to phoneme generation of Kupiec' s text input via 
character recognition of a scanned image to create words and phonemes that could populate Bird 
et al. 's lattice from the user's input because with that capability the user may add personalized 
annotation to the phoneme and word lattice that annotates the corpus. 



78. Claim 39 sets forth additional limitations similar to limitations set forth in claim 1 Bird et 
aL and Kupiec describe and make obvious the limitations as indicated there. Kupiec also 
describes further additional limitations as follows: 
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means for generating phonemes and/or words [at column 9, line 20, as the transcriber]. 
Bird et al. also describes further additional limitations as follows: 

means for generating the data for defining and associating [at section 4.2, as programming 
that takes advantage of the convenient links, hierarchies, and pointers]. 

79. Claim 40 sets forth additional limitations similar to limitations set forth in claim 2. Bird et 
al and Kupiec describe and make obvious the limitations as indicated there. 

80. Claim 41 sets forth additional limitations similar to limitations set forth in claim 3. Bird et 
al. and Kupiec describe and make obvious the limitations as indicated there. 

81 . Claim 42 sets forth additional limitations similar to limitations set forth in claim 4. Bird et 
al and Kupiec describe and make obvious the limitations as indicated there. 

82. Claim 43 sets forth additional limitations similar to limitations set forth in claim 5. Bird et 
al and Kupiec describe and make obvious the limitations as indicated there. 

83. Claim 44 sets forth additional limitations similar to limitations set forth in claim 6. Bird et 
aL and Kupiec describe and make obvious the limitations as indicated there. 



84. Claim 45 sets forth additional limitations similar to limitations set forth in claim 7. Bird et 
aL and Kupiec describe and make obvious the limitations as indicated there. 
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85. Claim 46 sets forth additional limitations similar to limitations set forth in claim 10. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 



86. Regarding claim 47, Kupiec [at column 29, line 66-column 30, line 2] describes that the 
phoneme data converting the text corpus to a phonetic index will be taken from a pronunciation 
dictionary. Thus the text data will define the parol of the one or more speakers of the 
pronunciation dictionary. 

However, Kupiec does not explicitly describe the parol of a plurality of speakers for the 
pronunciation dictionary. In addition, Kupiec does not provide a phonetic index for the parol of 
each speaker who is characterized in the pronunciation dictionary. 

Bird et al. [at section 3 .1 and Fig. 2] also describes a network embodiment of a phonetic 
index with similarities to the one of Kupiec , and B ird et al. also describes:: 

the speech data defines the parol of a plurality of speakers [at sections 3 .1 and 3 .2, as read 
speech of exemplary female speaker c jsp0' from the TIMIT corpus and the Parti tur speech signal 
fragment under discussion from the Verbmobil corpora]; and 

the data defines a separate phoneme and word lattice for the parol of each speaker [at 
sections 3.1 and 3.2 and Figs. 1 and 2, as the example file and acyclic graph for female speaker 
'jspO' from TIMIT and the example Partitur annotations lines expressed as in Figure 2]. 

Kupiec has text as a source file for the corpus, and so explicitly describes text recognition 
for conversion to the phoneme lattice, thus the individual pronunciations of each speaker of 
Kupiec 's phonetic data are not known. Bird et al. , on the other hand, provides an audio corpus of 
speech signals of each speaker, and uses the individual pronunciations to construct the annotation 
data. In view of the similarities between Bird et al. and Kupiec , an artisan seeking to generate 
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Kupiec ' s phonetic data would have found it obvious at the time of invention to provide the parol 
of individual speakers as Bird et al has done, rather than a pronunciation dictionary's phonetic 
data, because then the individual speaker could be identified, and the annotations data could 
indicate different speakers for different parts of a text corpus. 

87. Claim 50 sets forth additional limitations similar to limitations set forth in claim 12. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

88. Claim 51 sets forth additional limitations similar to limitations set forth in claim 13. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

89. Regarding claim 54, Bird et al. also describes: 

means for associating the annotation data with the data file [at section^C as offsets in 
time-function files that provide the time line to annotations including time references]. 

90. Claim 55 sets forth additional limitations similar to limitations set forth in claim 33. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

91. Claim 56 sets forth additional limitations similar to limitations set forth in claim 38. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 



92. Claim 57 sets forth additional limitations similar to limitations set forth in claim 38. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 
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93. Regarding claim 58, Kupiec also describes: 

a document scanner or a fax [at column 23, lines 24-28, as an input transducer for 
receiving an optically scanned document or other sequence of words]. 

94. Claim 59 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 34. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 

95. Claim 61 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 36. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 

96. Claim 62 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 37. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 

97. Claim 63 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 38. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 



98. Claims 64-71 each sets forth a method with limitations comprising the functionality 
associated with using the apparatus recited in claim 39-46, respectively. Because Bird et al. and 
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Kupiec describe and make obvious the similar limitations as indicated there, these claims thus are 
unpatentable accordingly. 



99. Claim 72 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 47, respectively. Because Bird et al. and Kupiec describe and 
make obvious the similar limitations as indicated there, these claims thus are unpatentable 
accordingly. 



100. Claims 75-76 each sets forth a method with limitations comprising the functionality 
associated with using the apparatus recited in claim 50-51, respectively. Because Bird et al. and 
Kupiec describe and make obvious the similar limitations as indicated there, these claims thus are 
unpatentable accordingly. 



101. Claim 79 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 54. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 



102. Claim 80 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 55. Because Bird et al. and Kupiec describe and make 
obvious the similar limitations as indicated there, this claim thus is unpatentable accordingly. 



103. Claims 81-83 each sets forth a method with limitations comprising the functionality 
associated with using the apparatus recited in claim 56-58, respectively. Because Bird et al. and 
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Kupiec describe and make obvious the similar limitations as indicated there, these claims thus are 
unpatentable accordingly. 



104. Regarding claim 85, Kupiec also describes: 

a phoneme and/or word lattice [at column 12-line 58-column 13, line 36, as the phonetic 
index represented as a network and orthographic vector]. 

Kupiec , however, does not explicitly describe the vector-network combination as one 

lattice. 

Bird et al. describes a network embodiment of a phonetic index with similarities to the one 
o f Kupiec . Thus, Bird et al. describes: 

(i) data defining nodes within a lattice [at section 3 . 1, as lines of numbers and labels in the 
corpus interpreting times as nodes in a directed, acyclic graph]; 

and data defining links connecting the nodes within a lattice [at section 3. 1, as lines of 
labels and numbers in the corpus interpreting labels as edges which connect nodes in a directed, 
acyclic graph]; 

(ii) data associating phonemes with the links [at section 3 J, as type of the edge label P for 
phonetic transcription]; and 

(iii) data associating words with links [at section 3 . 1, as type of the edge label W for 

word]. 

To the extent that Kupiec 's phonetic network and orthographic vector is not innately a 
phoneme and word lattice, it would have been obvious to one of ordinary skill in the art of 
searching queries through a database at the time of invention to include Bird et al. 's word and 
phoneme lattice framework with Kupiec' s concept of a phonetic network representing words in 
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the vector because that would simplify the annotation framework to a single structure and allow 
phoneme and/or word searching to be combined in Kupiec ' s search parameters. 

105. Claim 88 sets forth additional limitations similar to limitations set forth in claim 85. Bird 
et al. and Kupiec describe and make obvious the limitations as indicated there. 

106. Claim 96 is set forth alternatively including the limitations of claim 14. Bird et al. and 
Kupiec describe and make obvious those limitations as indicated there. Kupiec also describes 
additional limitations as follows: 

a data-carrier carrying processor-implementable instructions for controlling a processor to 
implement the method [at column 29, lines 39-49, as software on a digital computer to implement 
the embodiment]. 

107. Claim 97 is set forth alternatively including the limitations of claim 14. Bird et al. and 
Kupiec describe and make obvious those limitations as indicated there. Kupiec also describes 
additional limitations as follows: 

processor-implementable instructions for controlling a processor to implement the method 
[at column 29, lines 39-49, as software to implement the embodiment]. 



Bird et al and James et al 

108. Claim 1 1 is rejected under 35 U.S.C. 103(a) as being unpatentable over Steven Bird and 
Mark Liberman, "Towards a Formal Framework for Linguistic Annotations," ( Bird et al ), already 
of record, in view of D. A. James and S. J. Young, "A Fast Lattice-Based Approach to Vocabulary 
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Independent Wordspotting," 1994 IEEE Int. Conf on Acoustics, Speech, and Sig. Proc, 1994. 
ICASSP-94. 19-22 April 1994, vol. 1, pp. 1/3774/380 (James et al. V 



109. Regarding claim 1 1, Bird et al. [at section 1] describes that the annotation framework 
provides tools for searching databases. However, Bird et al. does not explicitly provide weights 
for the phonemes and words on the edges of the lattice that provide a measure of confidence in the 
search result. 

James et al. [at section 2] also describes a lattice framework and designs it to provide 
reliable search results quickly. So that the search result also returns a measure of its reliability, 
James et al. describes: 

a weighting for the phonemes and/or words associated with the links [at section 2, as a 
likelihood score for the phone labeling the edge]. 

Since both Bird et al. and James et al. annotate databases with similar lattices applied to 
searching the database, it would have been obvious to one of ordinary skill in the art of search and 
retrieval at the time of invention to add James et al. 's concept of likelihood for the edge labels to 
be able to provide weights when Bird et al. 's edge labels are matched during search because that 
would allow the user or further applications to assess how closely the results of the search 
compare to the search query. 



Bird et at and Kupiec and Garber 

1 10. Claims 15-19 and 25-29 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Steven Bird and Mark Liberman, "Towards a Formal Framework for Linguistic Annotations," 
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( Bird et al ) , already of record, in view of Kupiec [US Patent 5,500,920], and further in view of 
Garber et al. [US Patent 6,321,226], 



111. Regarding claim 1 5, Kupiec also describes: 

searching using the word data to identify similar words [at column 6, lines 35-39, as search 
by the IR subsystem using queries that comprise words]; and 

selecting one or more portions for further searching in response to the results of the word 
search [at column 27, lines 1-18, as use the relevant query results as a mini-corpus to facilitate 
understanding feedback inputs]. 

Although Kupiec also describes searching a phonetic network, Bird et al and Kupiec do 
not explicitly describe a phonetic search through portions selected by the word search. However, 
by describing the selected portions as a mini-corpus, all of Kupiec ? s teachings that apply to the 
corpus can be seen to apply to the selected portions. 

Garber [at column 8, lines 29-49] also describes word and phoneme searches, and Garber 
describes: 

searching using word data to identify similar words [at column 7, lines 30-3 1, as rough 
search identifying keyword phrases]; 

selecting one or more portions for further searching in response to the results of the word 
search [at column 7, lines 30-32, as identifying keyword phrases from the rough search]; 

searching selected portions using phoneme data of the query [at column 7, lines 32-35, as a 
subsequent step of matching phonetic forms of the individual words of the search phase]. 

Kupiec [at column 31, lines 26-28] suggests a coarse search followed by fine search. 
Garber [at column 7, lines 6-25] also indicates a rough search followed by a phoneme search. 
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Garber points out that some search strategies might return results that are not exact matches to the 
input query. In Garber' s example, the unwanted, initial results are words. It would have been 
obvious to one of ordinary skill in the art of searching queries through a database at the time of 
invention to include Garber 's concept of phonetic search of the results of an initial rough search 
because the subsequent phonetic search using only the initial results will identify which of the 
rough results are most relevant to the input query. 



1 12. Regarding claim 16, Kupiec also describes: 

the results of the word search are output to the user before the phoneme search is 
performed on the selected portions [at column 27, lines 1-18, as full text form of results are output 
to the user and user feedback on this can be used subsequently as a mini-corpus]. 



113. Regarding claim 17, Kupiec also describes: 

the phoneme search is only performed in response to further input by the user in response 
to outputting the results from the word search [at column 27, lines 1-18, as use the query results to 
facilitate understanding input that the use provides as feedback to the query results that are 
output]. 



1 14. Regarding claim 18, Kupiec also describes: 

the phoneme search is by identifying features in the phonemes sequence corresponding to 
the user's input query [at column 10, lines 29-60, as match phonetic indexes for the sequence of 
the use's utterance]; 
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and identify similar features in the data defining the phoneme lattice in the database [at 
column 14, lines 1-55, as match the sequence of phones to words in the phonetic index]. 

115. Regarding claim 1 9, Bird et al. also describes: 

each of the features represents a unique sequence of phonemes in the phoneme data of the 
user's inquiry [at section 5.3, as terms in the query define sequences of arcs for the query to 
particularize the association between the annotations and the files]. 

1 16. Claim 25 sets forth limitations similar to limitations set forth in claim 15. Bird et al., 
Kupiec, and Garber describe and make obvious the limitations as indicated there. Kupiec also 
describes further additional limitations as follows: 

means for searching using word data [at column 1 1, lines 1 1-12, as the IR subsystem]; 

means for selecting [at column 26, lines 66-67, as analyzer/evaluator]; and 

means for searching using phoneme data [at column 11, lines 1 1-12, as the IR subsystem]. 

117. Claim 26 sets forth additional limitations similar to limitations set forth in claim 16. Bird 
et al , Kupiec , and Garber describe and make obvious the limitations as indicated there. 

118. Claim 27 sets forth additional limitations similar to limitations set forth in claim 17. Bird 
et aL Kupiec , and Garber describe and make obvious the limitations as indicated there. 



1 19. Claim 28 sets forth additional limitations similar to limitations set forth in claim 18. Bird 
et aL Kupiec , and Garber describe and make obvious the limitations as indicated there. 
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120. Claim 29 sets forth additional limitations similar to limitations set forth in claim 19. Bird 
et aL Kupiec , and Garber describe and make obvious the limitations as indicated there. 

Bird et al and Kupiec and Garber and Rose 

121. Claims 20 and 30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Steven 
Bird and Mark Liberman, "Towards a Formal Framework for Linguistic Annotations," ( Bird et 
al), already of record, in view of Kupiec [US Patent 5,500,920] and Garber et al. [US Patent 
6,321,226], and further in view of Rose et al. [US Patent 5,870,740]. 

122. Regarding claim 20, neither Bird et al. nor Garber provide details of how input queries will 
be matched against the annotated data. Kupiec [at column 17, line 5-column 18, line 10] describes 
relevancy scoring of the results returned by the search, for example as the number of hits returned 
and the number of times that the hit appears in a document. However, Kupiec does not explicitly 
define a cosine similarity measure. 

Rose [at abstract] also describes searching queries through a database, and providing a 
measure to rank the relevancy of the results that are returned. Rose describes: 

a cosine measure to indicate the similarity between the data of the query and the data 
within the database [at column 6, lines 8-18, as the cosine similarity metric formed of parameters 
of the query and parameters of the document]. 

Rose [at column 3, lines 4-1 1] indicates that the cosine similarity metric is an existing 
statistical method to provide relevance ranking. At the time of invention, an artisan in the art of 
search and retrieval would have recognized Kupiec' s exemplary parameters for relevancy ranking 
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as parameters of the existing cosine metric, for example as described by Rose , It would have been 
obvious to one of ordinary skill in the art to include Rose 's concept of a cosine similarity metric to 
the similarity of Kupiec ' s phonetic matches between the queries and the database because Kupiec 
already uses parameters of the cosine metric as a relevancy score, and Rose points out that the 
cosine similarity metric can help to identify documents that are well-characterized by each query 
search term. 



123. Claim 30 sets forth additional limitations similar to limitations set forth in claim 20. Bird 
et aL Kupiec , Garber, and Rose describe and make obvious the limitations as indicated there. 



Bird et al and Kupiec and James et al 

124. Claims 48-49, 52-53, 73-74, and 77-78 are rejected under 35 U.S.C 103(a) as being 
unpatentable over Steven Bird and Mark Liberman, "Towards a Formal Framework for Linguistic 
Annotations," ( Bird et al. V already of record, in view of Kupiec [US Patent 5,500,920], and further 
in view of D. A. James and S. J. Young, "A Fast Lattice-Based Approach to Vocabulary 
Independent Wordspotting," 1994 IEEE Int. Conf on Acoustics, Speech, and Sig. Proa, 1994. 
ICASSP-94. 19-22 April 1994, vol. 1, pp. I/377-I/380 ( James et al. V 

125. Claim 48 sets forth additional limitations similar to limitations set forth in claim 1 1 The 
limitations are not explicitly described by Bird et al. or Kupiec ; however, Bird et al. , Kupiec , and 
James et al. describe and make obvious the limitations as indicated there. 
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126. Regarding claim 49, Bird et al. [at section 1] describes that the annotation framework 
provides tools for searching databases. However, Bird et al. does not explicitly provide weights 
for the phonemes and words on the edges of the lattice that provide a measure of confidence in the 
search result. 

James et al. [at section 2] also describes a lattice framework and designs it to provide 
reliable search results quickly. So that the search result also returns a measure of its reliability, 
James et al. describes: 

defining a weighting for the words identified within the phoneme data [at section 2, as 
returning the best path cumulative matching score for keyword phones]. 

Since both Bird et al. and James et al. annotate databases with similar lattices applied to 
searching the database, it would have been obvious to one of ordinary skill in the art of search and 
retrieval at the time of invention to add James et al. 's concept of likelihood for the edge labels to 
provide scores when Bird et al. 's edge labels are matched during search because that would allow 
the user or further applications to assess how closely the accumulated best-path word results of the 
search compare to the search query. 

127. Claim 52 sets forth additional limitations similar to limitations set forth in claim 1 1 . Bird 
et al. , Kupiec , and James et al. describe and make obvious the limitations as indicated there. 



128. Claim 53 sets forth additional limitations similar to limitations set forth in claim 49. Bird 
et al. , Kupiec , and James et al. describe and make obvious the limitations as indicated there. 
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129. Claim 73 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 48. Because Bird et aL Kupiec , and James et al. describe and 
make obvious the similar limitations as indicated there, this claim thus is unpatentable 
accordingly. 



130. Claim 74 sets forth a method with limitations comprising the functionality associated with 
using the apparatus recited in claim 49. Because Bird et al. , Kupiec , and James et al. describe and 
make obvious the similar limitations as indicated there, this claim thus is unpatentable 
accordingly. 



131. Claims 77-78 each sets forth a method with limitations comprising the functionality 
associated with using the apparatus recited in claim 52-53, respectively. Because Bird et al. and 
Kupiec describe and make obvious the similar limitations as indicated there, these claims thus are 
unpatentable accordingly. 



Kupiec 

132. Claims 90-92 are rejected under 35 U.S.C. 103(a) as being unpatentable over Kupiec [US 
Patent 5,500,920]. 



133. Regarding claim 90, Kupiec describes and makes obvious the claimed limitations as a 
whole recognizable to one versed in the art because Kupiec [at column 13, lines 26-29] describes 
an embodiment of phoneme data as annotation data, and Kupiec explicitly describes the following 
elements: 
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a medium for storing a data file [at column 29, lines 39-57, as a computer storing files]. 

To the extent that the data stored as Kupiec 's files are not audio data and annotation data, 
including phoneme data, corresponding to the audio data, it would have been obvious to one of 
ordinary skill in the art at the time that the invention was made to store any data in the fields of the 
computer's files that are described in Kupiec because the data does not functionally relate to the 
substrate of the computer media and merely labeling the data as audio data and annotation data, 
including phoneme data, corresponding to the audio data would have been obvious matter of 
design choice to provide a meaningful label, because a meaningful label given to data provides 
information to the user concerning the type of data that is stored. 



134. Regarding claim 91, Kupiec describes and makes obvious the claimed limitations as a 
whole recognizable to one versed in the art because Kupiec [at column 13, lines 26-29] describes 
an embodiment of phoneme data as annotation data, and Kupiec explicitly describes the following 
elements: 

a medium for storing a data file [at column 29, lines 39-57, as a computer storing files]. 

To the extent that the data stored as Kupiec ' s files are not video data, audio data, and 
annotation data, including phoneme data, corresponding to the audio data, it would have been 
obvious to one of ordinary skill in the art at the time that the invention was made to store any data 
in the fields of the computer's files that are described in Kupiec because the data does not 
functionally relate to the substrate of the computer media and merely labeling the data as video 
data, audio data, and annotation data, including phoneme data, corresponding to the audio data 
would have been obvious matter of design choice to provide a meaningful label, because a 
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meaningful label given to data provides information to the user concerning the type of data that is 
stored. 

135. Regarding claim 92, Kupiec describes and makes obvious the claimed limitations as a 
whole recognizable to one versed in the art because Kupiec [at column 13, lines 26-29] describes 
an embodiment of phoneme data as annotation data, and Kupiec explicitly describes the following 
elements: 

a medium for storing a data file [at column 29, lines 39-57, as a computer storing files]. 

To the extent that the data stored as Kupiec ' s files are not text data and annotation data, 
including phoneme data, corresponding to the text data, it would have been obvious to one of 
ordinary skill in the art at the time that the invention was made to store any data in the fields of the 
computer's files that are described in Kupiec because the data does not functionally relate to the 
substrate of the computer media and merely labeling the data as text data and annotation data, 
including phoneme data, corresponding to the text data would have been obvious matter of design 
choice to provide a meaningful label, because a meaningful label given to data provides 
information to the user concerning the type of data that is stored. 



Conclusion 

136. The following references made of record and not relied upon are considered pertinent to 
applicant's disclosure: 

Bahl et al. [US Patent 4,980,918] describes a graph of phonetic links that is searched to provide 
preliminary results and the portion for the preliminary results is searched again. 
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Bronson [US Patent 5,136,655] describes stored audio-video data that is played back into a voice 

recognition system and stored again with timing data and a word index. 
Kaplan [US Patent 5,721,939] produces a finite-state machine of text documents by optical 

character recognition of document images from a scanner. 
Liaguno et al. [US Patent 5,729,741] combines input text, text derived from input voice, and text 

derived from input still images, and text derived from input video in a searchable database. 
Pereira et al. [US Patent 5,781,884] produces a finite-state transducer with weighted phoneme 

links from orthographic representation of documents. 
Philippe Gelin and Chris J. Wellekens, "Keyword Spotting for Video Soundtrack Indexing," 1996 

IEEE Int. Conf. on Acoustics, Speech, and Sig. Proa, 1996. ICASSP-96. Conference 

Proceedings, vol. 1, pp. 299-302, 7-10 May 1996 describes searching a video soundtrack 

by keyword spotting in a phonetic probability lattice. 
Justin Zobel and Philip Dart, "Phonetic String Matching: Lessons from Information Retrieval," 

SIGIR Forum, Assoc. for Computing Machinery, New York, 1996, pp. 166-172 describes 

converting a file text to phonemes and retrieval by combining weighted results of phonetic 

matching and spelling matching of q-gram substrings. 



137. Any response to this action should be mailed to: 

Commissioner of Patents and Trademarks 
Washington, D C 20231 

or faxed to: 

(703)872-9314, (for formal communications intended for entry) 

Or: 
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(703)872-9314, (for informal or draft communications, and please label 
"PROPOSED" or "DRAFT") 

Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal 
Drive, Arlington, VA., Sixth Floor (Receptionist). 



138. Any inquiry concerning this communication or earlier communications from the examiner 
should be directed to Donald L. Storm, of Art Unit 2654, whose telephone number is 
(703)305-3941. The examiner can normally be reached on weekdays between 8:00 AM and 5:00 
PM Eastern Time. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Marsha D. Banks-Harold can be reached on (703)305-4379. Any inquiry 
of a general nature or relating to the status of this application or proceeding should be directed to 
the Technology Center 2600 Customer Service Office at telephone number (703)306-0377. 



Donald L. Storm 
Patent Examiner 

December 10, 2002 Art Unit 2654 



